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Abstract 


Organizations  are  increasingly  looking  for  guidance  on  what  it  takes  to  implement  Capability  Ma¬ 
turity  Model®  Integration  (CMMI®)  high  maturity  practices  and  how  to  sustain  their  momentum 
for  improvement.  As  high  maturity  organizations  work  to  improve  their  use  of  measurement  and 
analysis,  they  often  look  to  examples  of  successful  implementations  for  guidance.  In  response  to 
the  need  for  clarification  and  guidance  on  implementing  measurement  and  analysis  in  the  context 
of  high  maturity  processes,  members  of  the  SEI’s  Software  Engineering  Measurement  and  Analy¬ 
sis  (SEMA)  initiative  organized  a  workshop  at  the  2008  SEPG  North  America  conference  to  bring 
leaders  in  the  field  together  at  a  forum  on  the  topic.  Other  workshops  will  be  held  as  part  of  an 
ongoing  series  to  allow  high  maturity  organizations  to  share  best  practices  and  case  studies. 
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1  Introduction 


More  and  more  organizations  are  striving  for  and  reaching  high  maturity  status,  yet  there  is  still  an 
insufficient  shared  understanding  of  which  measurement  and  analysis  related  practices  are  appro¬ 
priate  for  high  maturity  organizations.  Although  Capability  Maturity  Model®  Integration 
(CMMI®)  provides  high-level  guidance,  some  organizations  struggle  to  find  an  effective  path  to 
high  maturity,  and  those  that  have  reached  it  must  persist  in  evolving  their  efforts  in  the  spirit  of 
continuous  improvement.  As  a  result,  organizations  are  increasingly  looking  for  guidance  on  what 
it  takes  to  reach  CMMI  high  maturity  status  and  how  to  keep  improving  once  they  get  there. 

1.1  Overcoming  Barriers  to  High  Maturity 

The  Software  Engineering  and  Measurement  and  Analysis  (SEMA)  initiative  at  the  Software  En¬ 
gineering  Institute  (SEI)  works  with  organizations  to  develop,  evolve,  and  evaluate  measurement 
and  analysis  practices.*  SEMA  researchers  have  identified  several  barriers  to  CMMI  high  maturity 
during  their  work  in  the  field.  In  this  section,  these  barriers  are  described  and  solutions  are  pre¬ 
sented. 

Examples  and  Case  Studies  Are  Needed 

SEMA  launched  two  new  measurement  training  offerings  in  2007  and  2008:  Improving  Process 
Performance  Using  Six  Sigma  (IPPSS)  and  Designing  Products  and  Processes  Using  Six  Sigma 
(DPPSS).  The  intent  of  the  courses  is  to  expand  the  use  of  statistical  modeling,  including  various 
forms  of  regression,  simulation,  and  probabilistic  modeling.  These  courses  teach  the  use  of  logis¬ 
tic  and  dummy  variable  regression  in  addition  to  traditional  simple  linear  multiple  regression  so 
practitioners  can  use  modeling  techniques  that  support  both  continuous  and  discrete  data  types. 

Students  attending  the  courses  sought  a  wider  array  of  industry  examples  showing  how  different 
kinds  of  process  performance  models  could  lead  to  better  performance  outcomes.  If  a  wider  set  of 
examples  is  collected,  future  course  updates  could  include  example  modules  that  are  more  closely 
related  to  the  domain  and  frame  of  reference  of  the  students. 

Also  in  recent  years,  a  number  of  CMMI  high  maturity  consultants.  Lead  Appraisers,  and  spon¬ 
sors  have  questioned  the  business  value  of  statistically  based  process  performance  models.  To 
accelerate  the  community  sharing  of  benefit  information  to  address  these  concerns,  SEMA  de¬ 
cided  to  collect  a  compelling  set  of  benefit  experiences  and  example  business  cases  for  the  devel¬ 
opment  and  use  of  process  performance  models. 

Misconceptions  About  Process  Performance  Models  (PPMs)  Need  to  Be  Dispelled 

During  client  work  by  the  SEMA  team  in  2005-2008,  it  became  clear  that  a  misconception  existed 
about  process  performance  models.  Most  clients  believed  that  the  chief  barrier  to  modeling  was 
the  need  for  advanced  knowledge  of  statistics.  However,  during  the  past  two  years,  students  in  the 


Additional  information  about  the  SEMA  initiative  is  available  at  http://www.sei.cmu.edu/sema/. 

See  http://www.sei.cmu.edu/products/courses/p49b.html  and 

http://www.sei.cmu.edu/products/courses/p56b.html  for  additional  information  about  these  courses. 
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IPPSS  and  DPPSS  courses  have  reaffirmed  that  the  domain  knowledge  used  to  identify  the  proper 
set  of  factors  (y’s  and  x’s)  remains  the  greatest  challenge,  not  statistical  knowledge.  The  courses 
include  job  aids  that  minimize  the  memorization  of  statistics  and  statistical  theory.  Using  these  job 
aids,  students  are  almost  unanimous  in  the  assessment  that  domain  knowledge  remains  the  great¬ 
est  challenge.  Students  still  gain  a  sufficient  understanding  of  statistics  to  recognize  and  avoid 
common  misuses  and  know  when  to  ask  for  help  from  coaches.  They  are  also  asked  to  seek  out 
mentors  in  the  workplace  who  possess  the  expertise  to  solve  real-time  problems  and  drive  com¬ 
pelling  business  improvements.  This  coaching  and  mentoring  structure  is  taken  from  the  Six  Sig¬ 
ma  realm  in  which  people  in  a  hierarchy  of  “belts”  coach  one  another.  Coaching  is  the  single  as¬ 
pect  that  has  best  enabled  the  successful  use  of  Six  Sigma  over  the  past  20  years. 

Adoption  of  Process  Performance  Models  Needs  to  Be  Accelerated 

SEMA  researchers  realized  community  adoption  of  process  performance  models  needed  to  be 
accelerated  to  meet  immediate  business  needs  and  show  return  on  investment  for  CMMI  business 
improvement.  Thus,  instead  of  waiting  the  projected  five  to  seven  years  for  statistically  based 
process  performance  models  to  become  more  widely  adopted,  SEMA  aims  to  help  the  community 
achieve  significant  adoption  in  the  next  two  to  three  years.  An  accelerated  schedule  will  also  be  in 
keeping  with  planned  CMMI  model  changes  and  the  rollout  of  additional  CMMI  constellations. 
Of  special  note,  discrete  event  simulation,  an  effective  modeling  approach,  has  already  been  wide¬ 
ly  adopted  by  the  services  community  to  predict  things  like  cycle  time,  workflow  bottlenecks, 
wait  times,  and  queue  lengths. 

Experienced  Coaches  and  Information  Sharing  Are  Needed 

Compared  to  the  size  of  the  CMMI  Lead  Appraiser  community,  the  size  of  the  CMMI  high  matur¬ 
ity  coaching  and  mentoring  community  is  very  small.  During  2007  and  2008,  it  became  apparent 
that  a  number  of  SEI  clients  who  wanted  to  pursue  CMMI  high  maturity  needed  hands-on  coach¬ 
ing  related  to  the  development  of  process  performance  models.  The  need  for  coaches  and  mentors 
knowledgeable  about  CMMI  high  maturity  topics  could  rapidly  exceed  the  need  for  appraisal  ser¬ 
vices,  and  SEI  Partners  offering  high  maturity  coaching  could  be  in  much  greater  demand. 

The  SEI  is  establishing  an  SEI  CMMI-Six  Sigma  Belt  certification  program  that  will  provide 
Black  Belt  and  Master  Black  Belt  coaches  via  an  SEI  Partner  list.  This  approach  will  provide  a 
venue  for  small-  to  medium-sized  organizations  to  network  and  learn  from  others  that  are  also 
developing  process  performance  models."* 

Lead  Appraisers  Need  Experience  Evaluating  High  Maturity  Measurement  Activities 

The  CMMI  High  Maturity  Lead  Appraiser  Oral  Exam  is  an  opportunity  for  appraisers  to  demon¬ 
strate  their  knowledge  of  required  topics,  discuss  professional  experiences,  and  show  their  under- 

^  Many  examples  of  the  use  of  discrete  event  simulation  in  the  services  community  can  be  found  at 
http://www.processmodel.com/resources/samplemodels.html 

http://www. allbusiness. com/3470945-1. html?query=%22discrete+event+simulation%22+services&x=0&y=0,  and 
http://search-www.isixsigma.com/cgi- 

bin/ss_query?related=0&keys=case+%2B%22discrete+event+simulation%22+%2Bservice&sitenbr=1 30985463. 
For  further  information,  see  Moving  Up  the  CMMI  Capability  and  Maturity  Levels  Using  Simulation  [Raffo  2008]. 

“  A  brief  description  of  this  program  is  available  in  an  SEI  Partner  Network  newsletter  [SEI  2008]. 
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standing  of  the  roles  and  responsibilities  of  a  High  Maturity  Lead  Appraiser.  During  the  first  year 
the  exam  was  given,  it  became  apparent  that  many  Lead  Appraisers  were  disadvantaged  in  evalu¬ 
ating  evidence  during  CMMI  High  Maturity  SCAMPIs.  Many  lacked  direct  experience  in  con¬ 
ducting  process  performance  modeling  and  had  not  observed  high  maturity  organizations  per¬ 
forming  process  performance  modeling.  As  a  result,  they  had  little  frame  of  reference  to  evaluate 
evidence  of  process  performance  modeling  during  the  SCAMPI  A’s.  From  this  perspective,  work¬ 
shops  would  illuminate  an  entire  landscape  of  modeling  that  participants  could  use  to  further  their 
professional  development  and  assist  clients  seeking  additional  guidance  on  process  performance 
modeling  best  practices. 

1.2  High  Maturity  Practices  Workshop  Series 

A  series  of  twice-yearly  SEI  workshops  has  been  planned  to  address  the  challenges  and  commu¬ 
nity  needs  described  in  Section  1 . 1  by  encouraging  organizations  to  share  their  experiences 
throughout  the  wider  community.  The  workshop  format  was  selected  to  allow  organizations  to 
share  lessons  learned  in  deployment,  adoption,  and  institutionalization  of  CMMI  process  perform¬ 
ance  baselines  and  models  with  the  goal  of  improving  the  practice  of  and  value  added  by  meas¬ 
urement  and  analysis  in  high  maturity  organizations. 
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2  High  Maturity  Workshop  Series  Kickoff 


2.1  Workshop  Participants  and  Goais 

The  focus  of  the  first  workshop  was  building  and  using  CMMI  process  performance  models.  Par¬ 
ticipation  was  limited  to  a  small  group  of  organizations  who  were  early  adopters  of  process  per¬ 
formance  models  and  baselines  and  was  by  invitation  only.  Representatives  from  Hill  Air  Logis¬ 
tics  Center,  Lockheed-Martin,  Northrop  Grumman,  and  Raytheon  attended. 

The  main  goals  of  the  workshop  were  to 

•  allow  CMMI  high  maturity  organizations  to  share  best  practices  and  case  studies 

•  identify  ways  to  develop  CMMI  high  maturity  measurement  and  analysis  practices  and  accel¬ 
erate  their  adoption 

•  enable  networking  among  practitioners 

2.2  Workshop  Structure 

The  workshop  was  scheduled  for  two  days  and  was  held  in  conjunction  with  the  SEPG  North 
America  conference  March  17-20,  2008,  in  Tampa,  Florida.  The  workshop  began  with  an  SEI 
presentation  summarizing  the  interpretation  of  process  performance  models  and  baselines  and  an 
overview  of  the  SEMA  CMMI  high  maturity  project.  This  presentation  is  available  at 
http://www.sei.cmu.edu/sema/presentations/hmworkshop.pdf 

Each  organization  gave  a  20-minute  presentation  summarizing  its  past  experiences  and  future 
plans  related  to  the  following  topics: 

•  barriers  faced 

•  lessons  learned  in  the  deployment,  training,  adoption,  and  institutionalization  of  CMMI  proc¬ 
ess  performance  baselines  and  models 

•  best  practices  and  examples  of  valid,  practical  methods  for  implementing  process  perform¬ 
ance  models  and  baselines 

•  data  quality  and  integrity  issues 

•  plans  for  modeling  over  the  next  three  to  six  months,  including  the  nature  of  the  performance 
outcomes  and  drivers  most  likely  to  be  investigated 

•  suggestions  for  subject  matter  to  include  in  future  SEI  state-of-the-practice  studies 

2.3  Summary  of  Presentations 
Barriers  Faced 

The  organizations  noted  that  they  faced  challenges  in  the  following  areas: 

1 .  establishing  the  value  of  developing  and  using  process  performance  models  (PPMs)  and 
baselines  (PPBs) 
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2.  convincing  project  managers  to  collect  new  measures  to  be  used  for  their  PPMs  and  PPBs 

3.  retaining  consistent  operational  definitions  as  the  scope  of  their  measurement  and  analysis 
activities  expand 

The  participants  expressed  a  strong  consensus  about  these  barriers.  As  with  any  new  initiative  or 
tool  adoption  requiring  significant  investment,  the  business  value  of  statistically  based  process 
performance  models  must  be  communicated.  Without  immediate  help  in  this  area,  many  partici¬ 
pants  felt  that  their  efforts  to  convince  management  and  the  organization  to  collect  additional 
measures,  with  consistent  operational  definitions,  would  be  an  uphill  battle.  Although  participants 
agreed  that  domain  knowledge  was  the  greatest  challenge,  there  remains  a  moderate  degree  of 
effort  to  create  models.  Several  participants  recounted  that  the  effort  to  create  individual  models 
took  several  weeks  or  months.  For  some,  this  was  unexpected  as  they  incorrectly  believed  that 
modeling  was  a  one-pass  approach  warranting  only  several  hours  of  effort. 

Lessons  Learned 

Among  the  lessons  learned,  the  workshop  attendees  noted  that 

•  useful  PPMs  and  PPBs  require  domain  and  statistical  knowledge.  Neither  alone  is  sufficient. 

•  coaching  and  mentoring  are  critical  elements  of  the  adoption  strategy  when  developing  and 
using  the  PPMs.  This  includes  guidance  on  possible  decisions  and  actions  associated  with  re¬ 
sults  from  PPMs  and  PPBs. 

Participants  echoed  the  need  for  domain  knowledge  in  developing  models.  Some  even  shared  ex¬ 
periences  in  which  statistical  experts  lacking  sufficient  domain  knowledge  created  models  that 
had  little  value  to  the  organization  and  its  projects.  For  this  reason,  many  participants  noted  that 
their  organizations  were  striving  to  involve  a  variety  of  domain  experts  in  developing  models. 

A  number  of  organizations  noted  that  their  modeling  experts  were  located  centrally  in  the  organi¬ 
zation  rather  than  at  the  project  level.  This  unfortunately  created  challenges  in  domain  relevancy 
and  made  it  difficult  to  have  a  project-level  focus  in  the  models  to  aid  actual  project  execution. 

All  of  the  organizations  represented  in  the  workshop  made  use  of  some  form  of  structured  coach¬ 
ing  and  mentoring.  Although  this  manifested  predominantly  in  the  form  of  Six  Sigma  belts,  sev¬ 
eral  of  the  organizations  used  other  methods  of  coaching  and  mentoring  for  modeling.  When 
questioned  by  the  SEl  group,  there  was  little  recognition  of  the  possible  need  for  upward  coaching 
and  mentoring.  The  SEl  group  shared  fhe  nofion  of  upward  mentoring  as  a  possible  improvement 
in  guiding  middle  and  upper  management  in  CMMl  high  maturity  practices,  specifically  in  the  use 
of  process  performance  modeling  to  manage  the  organization  and  projects.  For  mentoring  to  be 
successful,  process  and  behavioral  changes  are  needed  from  both  project  personnel  and  manage¬ 
ment.  For  example,  upper  management  may  benefit  from  coaching  to  enable  them  to  correctly 
interpret  and  use  at  the  organizational  level  the  results  of  the  analyses  conducted  at  the  project 
level. 
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Best  Practices 


Participants  discussed  what  they  considered  to  be  best  practices  and  tips  for  successful  implemen¬ 
tation  of  process  performance  models  and  baselines,  including 

•  providing  education  and  tools  to  support  modeling  and  analysis 

•  verifying  data  integrity  before  using  the  data  for  PPBs  and  PPMs 

•  performing  product  simulation  and  analysis  in  addition  to  process  simulation  and  analysis 

Almost  all  participating  organizations  provided  training  in  statistics  as  well  as  electronic  tools  for 
statistical  analysis  and  modeling.  Some  organizations  preferred  to  use  Microsoft  Excel  spread¬ 
sheets  for  analysis  and  Excel  add-ons  for  modeling.  Most  organizations  used  different  tools  to 
create  management  reports  because  the  statistical  tools  used  to  conduct  the  analyses  could  not 
produce  polished  graphs  for  management  slide  presentations. 

A  number  of  the  participating  organizations  instituted  a  series  of  significant  work  flow  checks  and 
balances  to  ensure  data  integrity  because  experience  had  taught  them  that  modeling  is  almost  im¬ 
possible  with  noisy  and  corrupt  data.  Finally,  the  SEl  team  was  surprised  at  the  degree  of  simula¬ 
tion  modeling  employed  by  several  of  the  participating  organizations.  This  might  have  resulted 
from  the  influence  of  systems  engineers  with  backgrounds  in  simulation  modeling. 

Data  Issues 

Data  quality  and  integrity  problems  that  are  encountered  at  lower  maturity  levels  continue  to 
threaten  the  potential  value  from  PPMs  and  PPBs.  Some  examples  include 

•  inconsistent  operational  definitions,  which  wreak  havoc  on  modeling  attempts 

•  data  collection  that  is  done  manually,  making  it  subject  to  human  error 

•  problems  aggregating  and  disaggregating  data 

•  missing  context  information  to  go  with  the  collected  data 

•  consistency  problems  arising  from  decentralized  databases 

These  issues  are  not  new.  The  SEl  has  heard  of  similar  issues  with  organizations  at  all  CMMl  ma¬ 
turity  levels.  As  organizations  make  greater  analytical  use  of  their  measures,  they  find  out  how 
many  data  integrity  issues  exist  in  their  data. 

Many  participants  recounted  the  need  to  revamp  their  measurement  and  analysis  programs  as  they 
progressed  up  to  CMMl  maturity  levels  4  and  5.  They  generally  agreed  that  their  measures  were 
not  at  the  proper  level  of  granularity  to  support  decision  making,  especially  at  the  project  level. 
They  also  recounted  experiences  in  which  aggregating  measures  from  across  the  organization  was 
hampered  by  the  lack  of  context  information  needed  for  proper  segmentation  and  stratification. 

Modeling  Plans 

Participating  organizations  planned  to  use  the  following  modeling  techniques  in  the  next  three  to 
six  months: 

•  Bayesian  methods  to  calculate  control  limits  during  statistical  management 

•  regression  analysis  to  model  and  predict  customer  satisfaction 
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•  measurement  system  evaluations  to  identify  the  degree  of  noise  in  data  due  to  the  measure¬ 
ment  process 

•  discrete  event  simulation  to  facilitate  Lean  Six  Sigma  improvements 

Again,  the  SET  team  was  surprised  by  the  variety  of  modeling  techniques  participating  organiza¬ 
tions  planned  to  use.  This  demonstrates  that  the  community  is  not  fearful  of  statistical  or  modeling 
techniques.  What  may  be  the  challenge,  as  discussed  earlier,  is  that  the  community  needs  to  learn 
how  to  more  effectively  apply  process  performance  modeling  in  support  of  project  execution.  In 
this  manner,  the  organization  will  have  institutional  learning  and  the  ability  to  affect  real-time 
execution  of  projects  towards  successful  outcomes. 

Subject  Matter  for  State-of-the-Practice  Studies 

Workshop  participants  suggested  the  following  topics  of  interest  for  future  SEI  state-of-the- 
measurement-practice  studies: 

•  adoption  and  use  of  measurement  and  analysis  related  to  high  maturity  practices,  particularly 
the  use  of  PPMs  and  PPBs 

•  balancing  statistical  and  domain  expertise  to  develop  and  sustain  the  value  and  use  of  PPMs 
and  PPBs 

•  bases  for  choosing  critical  subprocesses  to  place  under  statistical  control 

•  ways  in  which  to  develop  a  collection  of  useful  PPMs 

•  data  archeology  (i.e.,  creating  baselines  from  paper  records  for  previously  unmeasured  attrib¬ 
utes) 

•  data  quality  and  integrity 

The  SEI  team  reconfirmed  topics  for  the  2008  State  of  the  Measurement  Practice  survey  using  the 
suggestions  of  workshop  participants.  Due  to  the  need  for  an  unusually  detailed  survey  of  high 
maturity  organizations,  the  SEI  decided  to  conduct  two  annual  surveys  on  the  state  of  the  practice 
in  measurement:  1)  a  survey  for  the  general  community,  and  2)  a  survey  targeting  CMMI  high 
maturity  organizations.  In  this  manner,  each  subpopulation  could  be  given  information  pertinent 
to  their  perspectives  and  needs. 
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3  Future  Workshops 


The  SEMA  initiative  plans  to  hold  high  maturity  measurement  workshops  semi-annually  to  allow 
invited  attendees  to  continue  sharing  their  experiences  and  lessons  learned  in  the  adoption,  devel¬ 
opment,  and  use  of  measurement  and  analysis  in  high  maturity  settings. 

Participants  from  the  first  workshop  and  representatives  from  other  CMMI  high  maturity  organi¬ 
zations  were  invited  to  submit  proposals  for  presentations  at  the  next  workshop,  which  will  be 
held  in  Denver  immediately  following  the  CMMI  Technology  Conference  in  November  2008. 
Those  accepted  will  discuss  their  current  measurement  and  analysis  procedures  and  initial  results. 
SEI  experts  will  offer  additional  guidance  on  high  maturity  topics  and  present  pertinent  results 
from  the  2008  SEI  Slale  of  Ihe  MeasuremenI  Practice  survey. 

Planned  work  products  from  the  next  workshop  include 

•  thorough  case  descriptions  of  process  performance  models  and  their  outcomes  in  high  matur¬ 
ity  organizations 

•  break-out  working  session  reports  with  recommendations,  for  example  on  reducing  barriers  to 
effective  training,  staffing,  management  support,  the  alignment  of  modeling  to  business  goals, 
and  using  different  analytic  forms  of  modeling 

•  requirements  definitions  for  a  possible  SEMA  course  on  the  coaching,  adoption,  institution¬ 
alization,  and  evolution  of  CMMI  process  performance  models  and  baselines 

•  plans  for  a  coordinated  empirical  study  of  common  performance  outcomes  and  associated 
controllable  and  uncontrollable  drivers  of  those  outcomes 

Subsequent  workshops  will  be  open  to  a  larger  group  of  CMMI  high  malurily  organizations.  Or¬ 
ganizations  wishing  to  participate  in  fiiture  workshops  must  be  willing  to  document  and  share 
their  experiences  with  the  use  of  measurement  and  analysis  methods  in  relation  to  high  maturity 
practice.  To  ensure  high  value  workshops,  the  SEMA  team  will  continue  to  screen  submissions 
prior  to  accepting  an  organization’s  request  to  participate. 
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